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Preface 


The  Model  Execution  and  Evaluation  Tool  (MEET),  a  web-based  software  capability,  is  under 
development.  With  this  tool,  users  can  run  both  the  Mesoscale  Model  (Version)  5  (MM5)  and 
the  Weather  Research  and  Forecasting  (WRF)  model  and  generate  statistical  analyses.  The 
ability  to  store  statistical  results  in  an  XML  database  as  well  as  view  statistical  results  through 
the  web  interface  are  features  currently  being  integrated.  The  computer  platform  is  a  Linux 
cluster  with  16  dual-processor  nodes.  The  software  providing  the  parallelism  for  the  MM5  test 
runs  is  Scab  Message  Passing  Interface  (MPI).  Parallelism  in  the  case  of  WRF  is  created  by 
Message  Passing  Interface  Chameleon  (MPICH),  a  freeware  software  package. 

Given  this  significant  computing  capability,  a  four-member,  multi-analysis  MM5  model 
ensemble  is  being  run  and  analyzed.  The  four  members  consist  of  MM5  using  four  different 
coarser  resolution  models  for  initialization.  The  four  models  are  the  AViatioN  (AVN)  model 
(1°  horizontal  resolution),  Eta  (40  kilometer  horizontal  resolution),  the  Medium  Range 
Forecasting  (MRF)  model  (1°  horizontal  resolution)  and  the  Navy  Operational  Global 
Atmospheric  Prediction  System  (NOGAPS),  also  at  1°  horizontal  resolution. 

The  execution  time  for  a  four-member  MM5  model  “multi-analysis  ensemble”-using  three  grids 
(doubly  nested)  of  sizes  55x65x23,  36km  grid  spacing,  55x55x23,  12km  grid  spacing,  and 
85x85x23,  4km  grid  spacing,  with  a  forecast  period  of  24-h-is  approximately  two  hours.  The 
follow-on  model  to  MM5  is  the  WRF  model.  An  execute  capability  for  WRF  has  been  added  to 
MEET  and  analysis  algorithms  for  WRF  will  be  developed  as  well. 
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Summary 


The  model  execution  and  evaluation  process  has  many  components:  gathering  input  datasets, 
reconfiguring  model  domains,  running  a  model  or  model  ensemble,  as  well  as  gathering  truth 
datasets  to  statistically  analyze  the  accuracy  of  the  model  output.  These  tasks  can  be  very  time- 
consuming.  As  a  means  of  eliminating  a  significant  amount  of  this  manual  effort  and  to  allow 
the  user  to  concentrate  on  model  analysis,  a  Java  Server  Pages  based  software  module  called  the 
Model  Execution  and  Evaluation  Tool  (MEET)  is  being  implemented.  This  report  will  cover  the 
MEET  architecture  and  will  show  some  sample  results.  A  more  extensive  data  report  will  be 
written  later. 

Java  Server  Pages  were  chosen  for  this  web-based  tool  because  of  their  unique  ability  to  allow 
one  to  embed  java  code  within  either  HTML  or  XML.  This  is  a  powerful  paradigm  through 
which  one  can  both  generate  browser  display  as  well  as  control  external  programs,  such  as  the 
FORTRAN  model  codes,  modules  to  display  statistical  output,  etc. 

The  first  component  in  model  execution  is  to  obtain  the  large-scale  initialization.  What  options 
the  user  has  depends  on  which  model  the  user  wishes  to  run  and  in  what  mode:  standalone  or  as 
an  ensemble.  Within  MEET,  Mesoscale  Model  (Version)  5  (MM5)  can  be  run  in  either  a 
standalone  mode  or  as  a  four-member  multi-analysis  ensemble.  An  ensemble  is  being 
investigated  to  detennine  if  it  provides  more  accurate  parameters  in  the  model  output  than 
running  the  model  standalone. 

If  a  user  wishes  to  execute  a  Weather  Research  and  Forecasting  Model  (WRF)  run,  they  may  opt 
for  either  AViatioN  (AVN)  or  Eta  as  the  large-scale  initialization  data  whereby  an  anonymous 
FTP  is  automatically  initiated  to  get  the  appropriate  datasets  from  the  National  Center  for 
Environmental  Prediction  (NCEP). 

Once  the  model  runs  have  completed  and  the  truth  data  sets  have  been  acquired,  statistical 
analysis  may  begin.  For  model  surface  analysis,  the  user  will  have  access  to  mesonet  data  from 
the  inner-mountain  west  region.  For  upper  air  analysis,  raobs  from  worldwide  locations  are 
made  available  through  the  web  interface. 

After  the  statistical  analysis  is  complete,  the  results  will  be  stored  in  an  XML  database.  A  Java 
web  services  designed  specifically  for  MEET  enables  a  client  to  query  this  XML  database  for 
specific  statistics  on  the  model  output. 
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1.  Introduction 


The  Model  Execution  and  Evaluation  Tool  (MEET),  a  web-based  software  tool,  is  currently 
under  development.  The  MEET  has  recently  been  moved  from  a  single  processor  Linux 
platform  to  a  16-node  (each  node  with  a  dual  processor)  Linux  cluster.  Users  are  provided 
options  to: 

•  Execute  either  of  two  mesoscale  models:  Mesoscale  Model  (Version)  5  (MM5)  or  the 
Weather  Research  and  Forecasting  (WRF)  model.  MM5  can  be  run  using  either  archived 
Navy  Operational  Global  Atmospheric  Prediction  System  (NOGAPS)  data  (1997-present) 
or  as  a  four-member,  multi-analysis  ensemble  using  the  latest  0000  Universal  Time 
Coordinates  (UTC)  model  runs  of  AViatioN  (AVN),  Eta,  Medium  Range  Forecasting 
(MRF)  model,  and  NOGAPS  for  model  initialization.  WRF  currently  runs  using  a  single 
nest  only.  The  user  is  given  the  option  of  running  WRF  using  either  the  latest  0000  UTC 
AVN  or  0000  UTC  Eta  model  runs. 

•  Generate  statistical  output  from  the  model  outputs.  Statistical  analyses  for  temperature, 
relative  humidity,  and  the  u-  and  v-components  of  the  wind  are  available,  both  for  model 
output  near  the  surface  and  at  higher  levels.  The  truth  data  used  for  the  surface  statistical 
analysis  is  from  the  Mesowest  mesonet  located  in  Utah  and  the  adjoining  states.  For  the 
upper  air  statistical  analyses,  raobs  are  used  as  truth. 

•  Convert  the  statistical  output  files  to  XML  and  store  in  an  XML  database. 

•  View  the  statistical  analysis  data.  Bias  plots  and  scatterplots  of  temperature,  relative 
humidity,  and  the  u-  and  v-components  of  the  wind  can  be  created. 

To  ensure  that  the  best  meteorological  model  data  is  disseminated  for  Army  purposes,  model 
evaluation  is  carried  out  at  the  U.S.  Army  Research  Laboratory  on  an  ongoing  basis.  Model 
evaluation  has  many  facets: 

•  Data  collection-both  to  initialize  the  models  and  to  verify  them. 

•  Model  execution. 

•  Analysis  of  the  model  results-which  is  realized  both  in  statistics  and  graphical  plots. 

•  Storage  of  the  statistical  results  in  a  database  (not  always  done  but  is  beneficial  for 
historical  studies  or  in  looking  for  trends).  This  allows  a  client  to  query  the  database  for 
model  statistics.  Storage  of  the  model  output  itself  will  soon  be  quite  feasible  because  of 
rapid  increase  in  drive  capacities. 
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With  the  development  of  the  MEET,  all  of  these  components  will  be  available  through  a  web 
interface.  This  tool  is  designed  to  run  models  both  on  an  individual  basis  and  as  ensembles. 


2.  Methodology 


The  only  MM5  large-scale  initialization  model  that  has  a  historical  dataset  archive  readily 
available  is  NOGAPS.  Through  MEET,  the  user  has  the  option  of  running  MM5  using 
NOGAPS  datasets  from  1997  to  present.  In  addition,  MM5  can  be  run  using  real-time  AVN, 

Eta,  or  MRF  data  from  the  anonymous  FTP  site  at  NCEP.  The  WRF  has  been  designed  to  run 
using  either  AVN  or  Eta  model  data,  which  is  available  in  real-time  from  the  NCEP  anonymous 
FTP  site.  Ensembles  can  be  run  in  three  different  modes:  1)  multi-analysis;  2)  multi-model;  or  3) 
multi-model,  multi-analysis.  The  ensemble  currently  available  in  MEET  is  the  MM5  multi¬ 
analysis  ensemble  and  consists  of  four  members,  in  which  MM5  has  been  initialized  with  AVN, 
Eta,  MRF,  and  NOGAPS  data.  Multi-model  and  multi-model/multi-analysis  ensembles  will  be 
feasible  when  the  WRF  model  is  redesigned  to  allow  nesting  of  domains. 

The  web  interface  is  built  around  Java  Server  Pages  (JSP),  which  allows  java  code  to  be 
embedded  within  HTML  or  XML.  Through  the  JSP,  calls  are  made  to  various  programs  and 
scripts  to  carry  out  user  requests.  For  example,  if  the  user  requests  a  MM5  run  using  NOGAPS, 
the  MM5  FORTRAN  code  is  recompiled  to  account  for  the  configuration  the  user  requests  (i.e., 
the  number  of  nests).  Then  the  JSP  will  call  on  a  number  of  scripts  to  process  the  NOGAPS  data 
in  preparation  for  the  MM5  run  before  finally  starting  the  MM5  run.  Scab  MPI  provides  the 
parallelism  in  software  for  the  MM5  model  runs,  while  for  WRF,  Message  Passing  Interface 
Chameleon  (MPICH)  is  used. 

Statistics  may  be  generated  both  for  surface  analysis  and  upper  air  analysis.  The  surface  analysis 
through  MEET  is  currently  restricted  to  the  inner-mountain  western  U.S.  This  is  because  the 
only  freely  available  real-time  mesonet  FTP  data  source  is  from  Mesowest,  which  provides  a 
continuous  live  feed  of  surface  parameters.  Upper  air  data  analysis  may  be  carried  out  almost 
globally  with  the  availability  of  raobs  from  every  continent  through  a  University  of  Wyoming 
HTTP  site:  http://weather.uwyo.edu/upperair/sounding.html.  Currently  available  statistics  to 
delineate  the  accuracy  of  MM5  model  output  temperature,  u-  and  v-components  of  the  wind,  and 
relative  humidity  (which  is  actually  derived  from  other  model  parameters)  include: 

•  bias 

•  mean  error 

•  mean  absolute  error 

•  root  mean  square  error 
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•  ensemble  spread 

•  ensemble  dispersion 

•  rank  correlation 

Through  the  use  of  a  java-based  graphics  development  toolkit  called  Visualization  for  Algorithm 
Development  (VisAD),  routines  have  been  developed  by  the  author  to  display  model  outputs 
versus  truth  as  scatterplots,  providing  the  user  a  view  of  model  output  quality.  Bias  plots  through 
VisAD  routines  are  also  available. 

For  the  MM5  ensembles,  a  means  of  fusing  the  ensemble  members  is  being  investigated, 
whereby  each  ensemble  member  parameter  is  assigned  a  weight  based  on  the  parameter  accuracy 
over  the  last  24-h  forecast.  The  more  accurate  the  ensemble  member  parameter  is  for  that  period, 
the  higher  the  weight  it  will  be  assigned.  The  four  weights  of  a  particular  parameter  from  the 
four-ensemble  members  total  to  a  value  of  one.  The  ensemble  members  are  then  “fused” 
together  on  a  parameter- wise  basis  given  the  pre-derived  weights.  Although  sample  weight 
values  have  been  calculated  for  ensemble  runs,  the  final  fusion  process  has  not  been  tested. 

A  means  for  a  client  to  seamlessly  obtain  model  output  statistics  from  the  XML  database  on  the 
server  has  been  developed  through  the  application  of  a  Java  Web  Service.  A  web  service  is 
defined  as  the  ability  of  one  software  application  to  communicate  with  another  application  via 
web  protocols  without  direct  human  intervention.  With  MEET,  the  client  can  request  specific 
model  output  statistics  for  a  given  model,  output  parameter,  and  forecast  lead  time. 

The  ability  to  both  obtain  model  inputs  and  execute  the  mesoscale  models  (MM5  and  WRF) 
through  a  web  interface  has  been  built.  Furthennore,  the  user  is  able  to  obtain  “truth”  data  from 
the  Mesowest  mesonet  and  perform  statistical  analyses  on  the  MM5  output.  In  addition,  the 
ability  to  run  MM5  as  a  multi-analysis  ensemble  and  generate  statistics  on  the  ensemble  output  is 
available.  Lastly,  the  WRF  model  has  been  recently  incorporated  and  may  be  run  using  either 
AVN  or  MRF  model  data  for  large-scale  initialization.  Statistical  routines  to  analyze  WRF 
output  need  to  be  implemented  next. 
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3.  MEET  Architecture  and  Tools  Available 


3.1  The  Server  Side 

3.1.1  Ensemble  Modeling  Background 

Significant  computing  capability  available  with  a  32-processor  Linux  cluster  made  it  feasible  to 
examine  MM5  run  as  an  ensemble.  There  has  been  a  progression  from  deterministic  forecasting 
to  stochastic  dynamic  forecasting  to  ensemble  modeling.  In  deterministic  forecasting,  one  takes 
the  initial  conditions  and  integrates  the  model  equations  forward.  In  this  case,  the  initial 
conditions  are  assumed  to  be  the  true  state  of  the  atmosphere.  However,  this  cannot  be  since 
there  are  instrument  errors  and  possible  errors  in  interpolation  of  data.  With  stochastic  dynamic 
forecasting,  the  prognostic  equations  are  modified  by  adding  terms  to  account  for  the  uncertainty 
in  the  initial  conditions.  These  terms  are  carried  throughout  the  integration,  leading  to  a 
distribution  of  solutions. 

In  the  scheme  used  here,  MM5  is  run  as  a  multi-analysis  ensemble.  It  is  multi-analysis  in  the 
sense  that  it  is  initialized  with  four  different  coarser  scale  models:  the  AVN  model,  the  Eta 
model,  the  MRF  model,  and  the  NOGAPS  model.  Combining  the  ensemble  members  allows  for 
smoothing  of  features  that  are  in  disagreement  between  the  individual  forecasts  and  the  retention 
of  features  that  are  similar.  Gerding  and  Myers  have  proposed  fusing  the  ensemble  members  in 
the  following  way  (7): 

xf = ( £  wifcif  ( xjf 1  bif )) ;( £  wifcif  >  ( 1 ) 

7=1  7=1 


where 
/=  variable 
Xf=  fused  forecast 
Xjf=  input  forecast 
Cjf=  associated  confidences 
wjf=  weights 
bjf=  biases 

The  fused  forecast  is  the  sum  of  the  bias  adjusted  input  forecasts,  modified  by  some  weight  and 
confidence.  For  simplification,  the  bias  is  assumed  to  be  zero  and  the  confidence  assumed  to  be 
one.  The  weights  are  calculated  according  to  the  Newbold-Granger  methodology,  as  follows: 
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w  if  =  [(  }  (xtf  -  x  jf  ))]  2  / 

t  =  T  -v 

T  - 1  _  2  (2 

I  [(  I  -  Xkf  )] 

A:  t  =  T  —v 

where 

Xy=  truth  for  the  forecast 
Xjf=  input  forecast 
v=  some  time  window 

k  =  refers  to  the  respective  ensemble  members 

Because  each  weight  is  normalized  by  the  sum  of  the  errors  for  a  given  parameter  over  all  the 
ensemble  members,  the  weights  for  a  given  parameter  across  all  ensemble  members  sum  to  one. 

3.1.2  Generating  Model  Runs 

The  server  side  of  MEET  refers  to  the  actions  taking  place  on  the  32-processor  cluster.  A  user 
providing  input  to  the  web  interface  of  MEET  (fig.  1)  is  communicating  information  directly  to 
the  Java  Server  Pages  modules. 


MODEL  SELECTION 

C  MM5  Multi- Analysis  Ensemble  -  latest  OOIJT 

Statistic s/XML  Database 

AVN/ETAMRF/NOGAPS  data 

C  MM5  —  archived  NOGAPS  (also  portal  to  MEL 

O  Surface  data  analysis 

NO  GAPS  data  if  needed  for  an  ensemble  run) 

O  Upper  air  data  analysis 

C  WRF  (OOIJT  AVN) 

Submit  Query  |  Reset  | 

WRF  (OOUT  ETA) 

O  MM5(AVN)  and  WRE(AVN)  -  latest  OOUT  data 

MM5(ETA)  and  WKF(ETA)  -  latest  OOUT  data 

Submit  Query  |  Reset  | 

MM5 

WRF 

C  Surface  temperature  bias  plot  (Utah  area) 

O  Surface  temperature  bias  plot  (Utah  area) 

C  Upper  air  temperature  bias  plot 

r  Upper  air  temperature  bias  plot 

C  Temperature  Scatterplot  (surface  model  and  trutb  data-Utah) 

O  Temperature  Scatterplot  (surface  model  and  truth  data-Utah) 

C  Relative  Humidity  Scatterplot  (surface  model  and  truth  data-Utah) 

r  Relative  Humidity  Scatterplot  (surface  model  and  truth  data-Utah) 

Submit  Query  |  Reset  j 

Submit  Query  |  Reset  | 

Figure  1.  Initial  Web  Interface  to  MEET. 
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The  user  has  six  options  in  the  MODEL  SELECTION  frame: 

1 .  MM5  Multi-Analysis  Ensemble-latest  OOUT  AVN/Eta/MRF/NOGAPS  data 

2.  MM5-archived  NOGAPS 

3.  WRF(OOUTAVN) 

4.  WRF  (OOUT  Eta) 

5.  MM 5  (AVN)  and  WRF  (AVN)-latest  OOUT  data 

6.  MM5  (Eta)  and  WRF  (Eta)-latest  OOUT  data 

If  the  user  selects  option  1,  an  anonymous  FTP  session  to  NCEP  will  immediately  begin 
acquiring  the  latest  0000  UTC  valid  run  time  of  the  AVN  and  Eta  analysis,  the  3-h,  6-h,  9-h,  12- 
h,  15-h,  18-h,  21-h,  and  24-h  forecasts.  For  the  MRF,  the  latest  0000  UTC  valid  run  time 
analysis,  12-h,  and  24-h  forecasts,  will  be  acquired.  There  is  no  anonymous  FTP  site  available 
for  NOGAPS.  Instead,  the  user  should  select  MM5-archived  NOGAPS,  at  which  point  an  option 
to  request  NOGAPS  data  will  be  given.  The  latest  0000  UT  valid  run  time  analysis,  the  6-h,  12- 
h,  18-h,  and  24-h  forecasts,  should  be  obtained.  Once  all  of  the  data  sets  are  on  the  server,  an 
MPI  run  begins,  utilizing  the  32  processors.  The  model  configuration  consists  of  three  domains, 
with  two  inner  nests.  The  outer  domain  is  55x65x23  points  at  36km  grid  spacing.  The  first 
nested  domain  is  55x55x23  points  at  12km  grid  spacing  while  the  innermost  nest  is  85x85x23 
points  at  4km  grid  spacing.  Currently,  there  is  no  provision  to  set  the  coarse  domain  center 
because  surface  verification  data  is  only  available  from  the  inner-mountain  west  region,  so  the 
coarse  domain  is  centered  approximately  over  Salt  Lake  City,  Utah.  The  ability  for  the  user  to 
set  the  domain  center  will  likely  be  made  available  in  a  future  revision  when  new  verification 
data  sets  are  identified. 

Option  2  in  the  MODEL  SELECTION  frame  lets  a  user  select  any  NOGAPS  data  set  from  the 
Master  Environment  Library  (MEL),  dating  back  as  far  as  1997,  to  use  as  initialization  data  for 
MM5  (fig.  2). 

NOGAPS 


r  Request  NOGAPS 
r  Pickup  NOGAPS 
r  Upload  NOGAPS 
f  Setup  a  MM5  Run 

Submit  Query  |  Reset  | 


Figure  2.  Setting  Up  a  MM5  Run  Using  Archived  NOGAPS. 
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The  user  may  request  NOGAPS  data  (through  MEL)  and  pick  up  the  requested  NOGAPS  data 
from  a  Naval  Research  Laboratory  (NRL)  LTP  pickup  site  (fig.  3). 


GET  ARCHIVED  NOGAPS  DATA 

From  here  you  can  go  to  the  Master  Environmental  Library  homepage.  First  select  HTML  Data 
Query.  This  brings  up  a  new  HTML  page.  In  the  "keywords"  text  box  type  in  nogaps.  Select 
NFLMRY  as  the  data  source.  Another  page  of  html  will  now  come  up.  Here  you  will  select  the 
time  periods,  parameters  and  levels  of  interest. 

TO  MEL  site  | 


Figure  3.  Screen  to  Obtain  NOGAPS  Data. 

Once  the  data  is  on  the  system,  the  user  chooses  upload  nogaps  (fig.  4).  The  user  will  be  able  to 
browse  their  system  and  then  will  click  upload  to  move  the  data  onto  the  server. 

FILE  UPLOAD 

|D:\MEL_nrlm-1 021  B0-o .■  Browse...  |  upload  nogaps  | 


Figure  4.  File  Upload  Screen. 

Lastly,  the  user  selects  “set  up  a  mm5  run,”  which  allows  for  specifying  the  run-time  window, 
model  domain  center  point,  and  number  of  grids  (fig.  5). 
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MM5  -  ARCHIVED  NOGAPS 


Model  center  latitude:  | 

Model  center  longitude:  | 
Number  of  grid  domains:  ra 

Start  year:  1 1 997  -  | 
Start  month:  1 01  ^  | 
Start  day:  j  01  -] 
Start  hour:  1 00  -| 
End  year:  1 1997 
End  month:  j  01  ^  I 
End  day:  1 01  -I 
End  hour:  1 00  -| 

Submit  Query  |  Reset  | 


Figure  5.  User  Screen  to  Setup  for  a  MM5  Run. 

If  option  3  is  selected  in  the  MODEL  SELECTION  frame,  an  anonymous  FTP  begins  for  the 
latest  0000  UTC  valid  run  time  of  the  AVN  model  and  as  with  the  ensemble  in  option  1,  the  00- 
24-hr  forecasts  at  3  hour  intervals  are  obtained.  The  WRF  Standard  Initialization  (WRFSI) 
routines  are  called  next  to  interpolate  (both  horizontally  and  vertically)  the  AVN  data  onto  the 
WRF  grid  for  use  as  initial  and  lateral  boundary  conditions.  Finally,  the  WRF  model  is  ready  to 
be  run.  Parallelism  is  created  for  the  WRF  runs  by  the  use  of  Message  Passing  Interface 
Chameleon  (MPICH),  a  freeware  package. 

Option  4  mirrors  option  3  except  that  the  initial  and  lateral  boundary  conditions  are  set  using  Eta 
model  data.  Options  5  and  6,  although  not  implemented  yet,  will  allow  for  side-by-side 
comparisons  of  MM5  and  WRF,  using  AVN  and  Eta  as  initialization  data.  Once  WRF  has  a 
nesting  capability,  it  will  be  possible  to  have  side-by-side  comparisons  with  a  high-resolution 
inner  nest. 

3.1.3  Statistical  Generation 

The  second  frame  in  the  initial  MEET  web  interface  is  Statistics/XML  database.  Here,  the  user 
has  the  option  to  begin  either  surface  or  upper  air  analysis.  If  the  user  selects  Surface  analysis, 
the  options  are  to:  (1)  download  Mesowest  mesonet  data  from  U  of  Utah  anonymous  FTP  site; 

(2)  upload  this  data  to  the  server;  (3)  generate  the  surface  statistics  given  some  MM5  model 
output  and  the  newly  acquired  mesonet  data;  and  (4)  store  the  statistical  results  in  the  eXist  XML 
database  as  seen  in  figure  6. 
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SURFACE  ANALYSIS 


f*"  Get  Me  s  owe  st  me  sonet  data  for  Utah  and  surrounding  region  (current  to 
T-24h) 

C  Upload  mesowest  mesonet  data  to  server 
C  Start  generating  surface  statistics  (and  convert  output  to  XML) 

C  Store  XML  surface  statistics  in  eXist  database 

Submit  Query  |  Reset  | 


Figure  6.  Surface  Analysis  Options. 

The  other  option  is  to  analyze  the  higher  levels  of  the  model  output  using  Radiosonde 
observation  (raob)  data  as  truth.  When  the  user  selects  Upper  air  data  analysis,  the  options 
generally  mirror  the  surface  data  analysis  options,  except  that  the  truth  data,  raobs,  will  be 
obtained  via  http  to  a  University  of  Wyoming  Internet  site  (fig.  7). 

UPPER  AIR  DATA  ANALYSIS 


c  Get  raob  data  from  U  of  Wyoming 
C  Upload  raob  data  to  server 

C  Start  generating  upper  air  statistics  (and  convert  output  to  XML) 
r  Store  XML  upper  air  statistics  in  eXist  database 

Submit  Query  |  Reset  | 


Figure  7.  Upper  Air  Data  Analysis. 
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3.1.4  Display  of  Statistics 

The  bottom  two  frames  of  the  initial  MEET  web  interface  (fig.  8),  with  the  MM5  frame  (left) 
and  the  WRF  frame  (right),  allow  for  the  display  of  VisAD  generated  graphics  of  the  statistical 
analysis. 


MM5 

Surface  temperature  bias  plot  (Utah  area) 

^  Upper  air  temperature  bias  plot 
Temperature  Scatterplot  (surface  model  and  truth  data-Utah) 

C  Relative  Humidity  Scatterplot  (surface  model  and  truth  data-Utah) 

Submit  Quety  |  Reset  | 


WRF 

C  Surface  temperature  bias  plot  (Utah  area) 
r  Upper  air  temperature  bias  plot 

f  Temperature  Scatterplot  (surface  model  and  truth  data-Utah) 
r  Relative  Humidity  Scatterplot  (surface  model  and  truth  data-Utah) 

Submit  Query  |  Reset  | 


Figure  8.  Two  Frames  Where  a  User  May  View  Either  MM5  or  WRF  Statistics. 

In  figure  9,  the  scatterplot  is  shown  in  the  left  frame.  Model  temperature  is  represented  along  the 
x-axis,  while  the  y-axis  represents  (mesonet)  station  temperatures.  Here,  the  JSP  has  taken  the 
current  temperature  scatterplot  generated  and  placed  it  in  the  /pics  directory  of  the  JSP  tree.  The 
results  seen  are  the  surface  data  from  a  22-h  MM5-NOGAPS  ensemble  forecast,  36km  grid 
resolution.  Bilinear  interpolation  has  been  used  to  interpolate  the  model  grid  data  to  the  station 
location. 


WRF 

r  Surface  temperature  bias  plot  (Utah  area) 

O  Upper  air  temperature  bias  plot 
Temperature  Scatterplot  (surface  model  and  truth  data-Utah) 
Relative  Humidity  Scatterplot  (surface  model  and  truth  data-Utah) 

Submit  Queiy  |  Reset  | 


Figure  9.  Sample  Display  of  a  Temperature  Scatterplot. 
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3.1.5  Statistical  Measures  Used  in  MEET 

A  number  of  statistical  values  are  calculated  within  MEET,  including  some  specific  for 
determining  the  accuracy  of  the  ensemble  forecasts  (2).  They  are: 

1.  Bias 

yi  -  °i  (3) 

where 

Vt  =  model  value 
Oj  =  truth  value 

This  measure  is  used  to  discover  whether  the  model  has  tendencies  to  either  over  or 
underestimate  any  variables. 

2.  Mean  error 

ME  =  l/n'Z(yi~oi )  (4) 

i=l 

where 

Vi  =  model  value 
Oj  =  truth  value 
n  =  number  of  values 

3.  Mean  absolute  error 

MAE  =  \/ n  Zb/  -  Oj  (5) 

i= 1 

where 

Vi  =  model  value 
Oi  =  truth  value 
n  =  number  of  values 
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4.  Root  mean  square  error 


RMSE  =  [l/n'£j(yi-oi)2]1'2  (6) 

i=i 

where 

Vi  =  model  value 
Oj  =  truth  value 
n  =  number  of  values 


5.  Ensemble  spread 


SP(f)  = 


1  /NUfij-fij) 


n= 1 


n= 1 


I  J 


1=1  y=l 


(7) 


where 

/=  forecast  variable 
N  =  number  of  ensemble  members 
i,  j  =  indices 
/  =  number  of  rows 
J  =  number  of  columns 

(Note:  The  overbar  with  the  i,  j  indices  indicates  an  average  over  the  entire  domain.) 

The  ensemble  spread  is  the  mean  squared  deviation  of  the  component  members  from  their 
collective  mean  and  provides  a  measure  of  the  uncertainty  with  the  forecast.  Low  spread 
forecasts  will  be  more  predictable.  Conversely,  high  spread  forecasts  tend  to  be  less  predictable 
because  no  one  forecast  has  a  high  probability  of  verifying  (3). 
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6.  Ensemble  dispersion 

disp  =  ^2[\  -  r(f,v)]sd(f)sd(v)  (8) 

where 

r  =  rank  correlation 
sd  =  standard  deviation 
/=  forecast  variable 
v=  corresponding  station  value 

Dispersion  error  is  an  indication  of  phase  errors  in  the  forecast  as  opposed  to  amplitude  errors. 

7.  Rank  correlation 

r  =  X*Wl*2X/  (9) 

where 

x  =  model  value 
y  =  corresponding  station  value 

The  rank  correlation  provides  a  measure  of  the  linear  relationship  between  2  parameters. 


3.1.6  Conversion  of  Statistics  Output  to  XML  and  Storage  in  XML  Database 

In  order  to  provide  a  universal  format  for  the  output  statistics  and  also  allow  them  to  be  stored  in 
an  XML  database,  java  routines  were  written  to  convert  both  surface  and  upper  air  statistics  to 
XML  format.  The  XML  format  files  are  not  viewable  in  the  browser  until  their  conversion  to 
HTML  format.  Java  API  for  XML  Processing  (JAXP)  objects,  such  as  StreamSource  and 
StreamResult,  are  used  to  perform  this  conversion  ( 4 ).  An  example  of  such  a  conversion  is 
shown  in  figure  10.  Here,  a  “dummy”  raob  file  has  first  been  converted  to  XML  by  the  Java 
routines.  Next,  the  JAXP  objects  apply  XML  stylesheets  to  the  XML. 
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Model  Evaluation  Tool 

MODEL:  MM5-MRF 


Lead  Time:  0.0 
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Figure  10.  Sample  Output  of  a  Conversion  from 
XML  to  HTML  Using  JAXP  Routines 
and  XML  Stylesheets. 


3.2  The  Client  Side 

A  java  web  service  (JWS)  has  been  built  so  a  client  can  access  the  XML  database  on  the  server 
to  obtain  a  particular  model  statistic.  A  web  service  gives  the  ability  of  one  software  application 
to  communicate  with  another  software  application  via  web  protocols  without  the  need  for  direct 
human  intervention.  The  protocol  used  for  messages  is  the  Simple  Object  Access  Protocol 
(SOAP).  Because  the  client  and  server  communicate  back  and  forth  with  SOAP -wrapped  XML 
messages,  platform  independence  is  achieved.  For  this  effort,  the  third  generation  SOAP 
routines  from  Apache  software,  assembled  in  a  package  called  “Axis,”  were  used.  One  tool  that 
ships  with  Axis  is  tcpmon,  a  GUI  that  allows  the  user  see  what  is  happening  “on  the  wire” 
between  client  and  server.  An  example  of  its  use  is  shown  in  figure  1 1 .  The  top  half  of  the  GUI 
displays  the  client  activity.  Here,  the  client  resident  on  a  Sun  Blade  100  has  requested  from  the 
XML  database  on  the  server,  a  Linux  PC,  the  0-hr  lead  time  forecast  for  the  temperature  bias  of 
the  MM5-MRF  ensemble  member  forecast  valid  May  12,  2002  0000  UTC. 
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Content-Length:  652 

<?xml  version="1.0"  encoding="UTF-8"?> 

<soapenv  Envelope  xmlns:soapenv="http: //schemas,  xmlsoap.org/soap/envelope/"  xmlns:xsd="http://www.  w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 

<soapenvBody> 

<nsl:getEnsembleErrorExist  soapenvencodingStvte  “"http://schemas.xmlsoap.org/soap/encoding/”  xmlns:nsl="urn:ensemble.exist"> 
<argO  xsi:tvpe="xsd:  string"  >mm5-mrf</argO> 

<argl  xsi:tvpe ="xsd:  string"  >2  003 -05-12: 00:00: 00</argl> 

<arg2  xsi:type="xsd:string">bias</arg2> 

<arg3  xsi:type="xsd:string">0</arg3> 

<arg4  xsi:type="xsd:  string"  >t</arg4> 

</nsl:getEnsembleErrorExist> 

</soapenv.Body> 

</soapenv  Envelope  > 


<?xml  version="1.0"  encoding="UTF-8"?> 

<soapenv Envelope  xmlns:soapenv=“http: //schemas,  xmlsoap.org/soap/envelope/"  xmlns:xsd="http://www. w3.org/2001/XMLSchema" 
|xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 

<soapenvBody> 

<nsl:getEnsembleErrorExistResponse  soapenvancodingSt\1e  “"http://schemas.xmlsoap.org/soap/encoding/"  xmlns:nsl="urn:ensemble_exisf’> 
<getEnsembleErrorExistReturn  xsi:tvpe="xsd:  string"  >  Sit;  existtext  exist:  id  =  &quot;2744&quot; 
exist:  source=aquot;/db/sk/surface_stats/different_ convert_surface.stats.to_ xml.  xml&quot;  xmlns:  exist =&quot; 
http: //exist.  sourceforge.net/NS/exist&quot;&gt;-9. 13  4689&lt;/exist:text&gt; 

</getEnsembleErrorExist  Return  > 

<  /ns  1:  getEnsembleErrorExistResponse  > 

</soapenvBody> 

</soapenv  Envelope  > 


Figure  11.  tcpmon  GUI  Showing  Communication  Between  a  Client,  Requesting  Statistics  from  an  XML  Database, 
and  the  Server  Response. 

The  response  sent  from  server  to  client  is  seen  in  the  bottom  half  of  the  tcpmon  graphical  user 
interface  (GUI).  After  traversing  the  folder  tree  in  the  XML  database,  the  java  web  service  finds 
the  value,  -9.134689,  in  the  “surface_stats”  folder. 

Figure  12  shows  a  screen  shot  of  the  GUI  being  developed,  which  will  provide  the  client  a  direct 
method  for  utilizing  the  java  web  service  to  query  for  model  statistics.  Options  to  query  the 
database  for  temperature,  wind  (root  mean  square  vector  error),  and  relative  humidity  statistics 
are  available.  This  GUI  is  written  using  Java  2  Standard  Edition  (J2SE)  routines. 


d 

Ensemble  Clientjava  Web  Service 

Client  About 

(•>  Temperature  root  mean  square  error 
O  Wind  root  mean  square  vector  error 
O  Relative  Humidity  root  mean  square  error 

For  Model  or  Ensemble: 

(•)  mm5-avn  O  mm5-eta  O  mm5-mrf  G  mm5-ngm  O  mm5- nogaps  O  ensemble 

At  Hour  0atest  00  UTC  model  run): 

<•>  0  Ol  02  03  04  05  06  07  08  09  OlO  Oil 

O  12  O  13  O  14  O  15  O  16  O  17  O'  18  O  19  O  20  O  21  O  22  O  23 

returned  value:  3.0  C 

Initiate  Model  run(s):  □  mm5  avn  □  mm5  eta  □  mm5  mrf  □  mm5  ngm  □  mm5  nogaps  □  Ensemble 


Figure  12.  GUI  Which  Allows  Client  to  Access  XML  Database  on  Server. 
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4.  Statistical  Results 


4. 1  MM5  Ensemble  Results 

The  ensemble  members  for  this  study  consisted  of  MM5  initialized  with  data  from  four  different 
models: 

1 .  AVN  (1°  horizontal  resolution) 

2.  Eta  (40km  horizontal  resolution) 

3 .  MRF  ( 1 0  horizontal  resolution) 

4.  NOGAPS  (1°  horizontal  resolution) 

Next,  some  preliminary  analysis  is  presented  to  indicate  trends  seen  to  date. 


4.2  Surface  Analysis 

Following  is  a  table  showing  samples  of  Newbold-Granger  weights  obtained  for  an  ensemble  run 
using  initialization  data  from  12  May  2003.  The  weights  apply  to  a  surface  data  analysis.  The 
truth  data  is  the  Mesowest  mesonet  data.  Bilinear  interpolation  is  applied  to  interpolate  the 
model  grid  to  the  station  locations.  As  discussed  earlier,  adding  the  weights  for  a  given 
parameter  across  the  members  yields  a  value  of  one.  Also,  a  higher  weight  is  reflective  of  the 
ensemble  member  producing  more  accurate  output.  The  trend  seen  in  this  table  is  fairly  typical. 
The  MM5-AVN,  MM5-Eta,  and  MM5-MRF  members  are  generally  close  in  their  predictions, 
while  the  MM5-NOGAPS  member  tends  to  lag,  especially  in  temperature  prediction. 

Table  1.  May  12,  2003  Ensemble  Run-36  km  Grid  Spacing. 


MM5-AVN 

MM5-Eta 

MM5-MRF 

MM5- 

NOGAPS 

T 

0.30 

0.29 

0.30 

0.11 

U-COMP 

0.25 

0.28 

0.26 

0.21 

V-COMP 

0.26 

0.28 

0.26 

0.20 

RH 

0.26 

0.23 

0.27 

0.24 

4.3  Upper  Air  Analysis 

For  the  upper  air  analysis,  raobs  from  the  University  of  Wyoming  Internet  site  serve  as  truth. 
First,  the  MM5  utility,  “interpb,”  is  used  to  convert  the  MM5  model  output  from  sigma  levels  to 
pressure  levels.  Then  an  AWK  script  extracts  the  mandatory  pressure  levels  from  the  raob 
report.  Next,  at  each  pressure  level,  bilinear  interpolation  is  perfonned  to  interpolate  the  model 
grid  data  to  the  raob  locations.  The  area  of  study  is  Utah  and  the  surrounding  region.  For  the 
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outer,  coarse  grid,  nine  raobs  typically  were  available.  For  the  inner  nest,  three  raobs  typically 
were  available,  and  for  the  finest  grid,  one  raob  was  available. 

Preliminary  results  indicate  that  for  temperature,  increasing  grid  resolution  had  only  a  minor 
effect,  and  there  were  only  minor  differences  between  ensemble  members.  The  typical  mean 
absolute  error  was  on  the  order  of  1-2°  (C),  except  near  the  model  top  of  lOOmb.  Raising  the 
model  top  to  50mb  should  reduce  these  errors.  For  relative  humidity,  however,  increasing  grid 
resolution  did  provide  more  accurate  output,  and  the  MM5-Eta  ensemble  member  performed 
best.  Mean  absolute  errors  for  relative  humidity  ranged  from  two  percent  to  35  percent.  For  the 
u-component  of  the  wind,  again,  increasing  model  grid  resolution  is  beneficial.  Mean  absolute 
errors  for  the  u-component  of  the  wind  ranged  anywhere  from  near  zero  to  3.5  m/s. 


5.  Conclusions 


A  prototype  Model  Execution  and  Evaluation  Tool  has  been  built.  It  allows  users  to:  (1)  access 
model  initialization  data;  (2)  execute  MM5  as  a  standalone  or  as  a  multi-analysis  ensemble,  as 
well  as  execute  WRF;  (3)  access  truth  data  for  surface  analysis  (mesonet  data)  for  the  Utah 
region  and  for  upper  air  analysis  (raobs  worldwide);  (4)  generate  statistics  for  MM5  as  a 
standalone  or  as  an  ensemble;  (5)  store  these  statistics  in  an  XML  database;  (6)  display  statistics 
using  VisAD  routines  as  bias  plots  and  scatterplots,  and  (7)  access  the  server  from  some  client 
system  to  query  for  statistical  results. 

Work  remains  to  be  done.  Generating  a  multi-model  ensemble  with  the  inclusion  of  WRF  can 
soon  be  done  with  the  addition  of  nesting  to  WRF.  Also,  analyses  need  to  be  carried  out  to 
quantify  any  accuracy  gained  by  using  an  ensemble  as  opposed  to  a  single  model. 
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Acronyms 


API 

Automated  Programming  Interface 

ARL 

Army  Research  Laboratory 

AVN 

Aviation  (Model) 

FORTRAN 

Formula  Translator 

FTP 

File  Transfer  Protocol 

GUI 

Graphical  User  Interface 

HTML 

Hypertext  Markup  Language 

J2SE 

Java  2  Standard  Edition 

JAXP 

Java  API  for  XML  Processing 

JSP 

Java  Server  Pages 

JWS 

Java  Web  Service 

MEET 

Model  Execution  and  Evaluation  Tool 

MEL 

Master  Environment  Library 

MM5 

Mesoscale  Model  (Version)  5 

MPI 

Message  Passing  Interface 

MPICH 

Message  Passing  Interface  Chameleon 

MRF 

Medium  Range  Forecasting  (Model) 

NCEP 

National  Center  for  Environmental  Prediction 

NOGAPS 

Navy  Operational  Global  Atmospheric  Prediction  System 

Raob 

Radiosonde  observation 

SOAP 

Simple  Object  Access  Protocol 

UTC 

Universal  Time  Coordinates 

VisAD 

Visualization  for  Algorithm  Development 

WRF 

Weather  Research  and  Forecasting  model 
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WRFSI 

XML 


WRF  Standard  Initialization 
extensible  Markup  Language 
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49170  PROPAGATION  PATH 
SAN  DIEGO  CA  92152-7385 

METEOROLOGIST  IN  CHARGE  1 

KWAJALEIN  MISSILE  RANGE 
PO  BOX  67 

APO  SAN  FRANCISCO  CA  96555 
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NCAR  LIBRARY  SERIALS 
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1000  ARMY  PENTAGON  ROOM  2E383 
WASHINGTON  DC  20310-1067 
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ATTN  DR  E  DUTOIT 
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